Time shifting extended reality media

ABSTRACT

In one example, a method includes presenting an extended reality (XR) experience to a plurality of user devices, wherein the presenting includes presenting a time control along with an XR stream, receiving a first signal from a first user device of the plurality of user devices via the time control, wherein the first signal indicates that a user of the first user device wishes to shift the XR stream to a first point in time that is different from a time that is currently being rendered in the XR stream, and presenting a personal XR environment to the first user device in response to the first signal, wherein the personal XR environment presents the first point in time in the XR stream to the first user device without changing a time point of the XR stream that is currently being presented to other user devices of the plurality of user devices.

This application is a continuation of U.S. patent application Ser. No.17/105,463, filed Nov. 25, 2020, now U.S. Pat. No. 11,481,983, which isherein incorporated by reference in its entirety.

The present disclosure relates generally to extended reality (XR) media,and relates more particularly to devices, non-transitorycomputer-readable media, and methods for time shifting extended realitymedia.

BACKGROUND

Extended reality (XR) is an umbrella term that encompasses various typesof immersive technology in which the real-world environment is enhancedor augmented with virtual, computer-generated objects. For instance,technologies such as virtual reality (VR), augmented reality (AR), andmixed reality (MR) all fall within the scope of XR. XR technologies maybe used to enhance entertainment experiences (e.g., gaming, movies, andthe like), educational and/or professional development (e.g., trainingsimulations, virtual meetings, and the like), and travel (e.g., virtualor guided tours of museums and historic sites, and the like).

SUMMARY

In one example, a method performed by a processing system including atleast one processor includes presenting an extended reality experienceto a plurality of user devices, wherein the presenting includespresenting a time control along with an extended reality stream,receiving a first signal from a first user device of the plurality ofuser devices via the time control, wherein the first signal indicatesthat a user of the first user device wishes to shift the extendedreality stream to a first point in time that is different from a timethat is currently being rendered in the extended reality stream, andpresenting a personal extended reality environment to the first userdevice in response to the first signal, wherein the personal extendedreality environment presents the first point in time in the extendedreality stream to the first user device without changing a time point ofthe extended reality stream that is currently being presented to otheruser devices of the plurality of user devices.

In another example, a non-transitory computer-readable medium storesinstructions which, when executed by a processing system, cause theprocessing system to perform operations. The operations includepresenting a time control along with an extended reality stream,receiving a first signal from a first user device of the plurality ofuser devices via the time control, wherein the first signal indicatesthat a user of the first user device wishes to shift the extendedreality stream to a first point in time that is different from a timethat is currently being rendered in the extended reality stream, andpresenting a personal extended reality environment to the first userdevice in response to the first signal, wherein the personal extendedreality environment presents the first point in time in the extendedreality stream to the first user device without changing a time point ofthe extended reality stream that is currently being presented to otheruser devices of the plurality of user devices.

In another example, a device includes a processor and acomputer-readable medium storing instructions which, when executed bythe processor, cause the processor to perform operations. The operationsinclude presenting a time control along with an extended reality stream,receiving a first signal from a first user device of the plurality ofuser devices via the time control, wherein the first signal indicatesthat a user of the first user device wishes to shift the extendedreality stream to a first point in time that is different from a timethat is currently being rendered in the extended reality stream, andpresenting a personal extended reality environment to the first userdevice in response to the first signal, wherein the personal extendedreality environment presents the first point in time in the extendedreality stream to the first user device without changing a time point ofthe extended reality stream that is currently being presented to otheruser devices of the plurality of user devices.

BRIEF DESCRIPTION OF THE DRAWINGS

The teachings of the present disclosure can be readily understood byconsidering the following detailed description in conjunction with theaccompanying drawings, in which:

FIG. 1 illustrates an example network related to the present disclosure;

FIG. 2 illustrates a flowchart of a method for presenting an extendedreality experience with time shifting capabilities, in accordance withthe present disclosure; and

FIG. 3 depicts a high-level block diagram of a computing devicespecifically programmed to perform the functions described herein.

To facilitate understanding, identical reference numerals have beenused, where possible, to designate identical elements that are common tothe figures.

DETAILED DESCRIPTION

In one example, the present disclosure enhances XR media by enablingtime shifting with the XR media. As discussed above, XR technologiessuch as virtual VR, AR, and MR may be used to enhance entertainmentexperiences (e.g., gaming, movies, and the like), educational and/orprofessional development (e.g., training simulations, virtual meetings,and the like), and travel (e.g., virtual or guided tours of museums andhistoric sites, and the like). XR information can be presented inmultiple sensory modalities, including the visual, auditory, haptic,somatosensory, and olfactory modalities. As such, XR can be used toenhance a user's enjoyment of a media by making the media experiencemore immersive.

Some XR media streams allow users to join an XR experience that isalready in progress. For instance, a user may join a multi-player videogame that other users have already been playing for an hour, or mayarrive late to a virtual museum tour. Although the user may be able tojoin the XR stream mid-way through the XR experience, the user istypically unable to access the portions of the XR experience thatoccurred before the user joined. Thus, the user may miss out on part ofthe XR experience. In some cases, this may lead to confusion ordisorientation on the part of the user. For instance, the portions ofthe XR experience that the user missed may have been critical tounderstanding the later portions of the XR experience that the user isnow present for. Moreover, the users who are already present in the XRexperience may not notice that a new user has joined the XR experience,which may make it difficult for the joining user to fully participate inthe XR experience.

Examples of the present disclosure provide a way for users of XRexperiences to time shift through the XR experience. More particularly,a user who has joined an XR experience may be able to move backwardthrough an XR stream to experience portions of the XR stream thatoccurred before the user joined. The user may also be able to moveforward through the XR stream up to a current time point in the XRstream. Where multiple users are participating in the same XRexperience, a personal XR environment may be provided for a user whowishes to move forward or backward in the XR stream, so that the usercan time shift through the XR experience on demand, without affectingother users' experiences. For instance, a user who has just joined anin-progress XR experience may move backward through the XR stream to seewhat he missed, while the other users may continue to experience thecurrent XR stream. Thus, examples of the present disclosure provide atemporary XR space for local time control.

The personal XR environment may also be used to acclimate joining usersto an in-progress XR experience and/or to ease a user's exit from the XRexperience. This may smooth the integration of the new user into the XRexperience. Further examples of the present disclosure may provide anotification to existing users of an XR experience when a new user joinsthe XR experience and/or enters a personal XR environment in order totime shift through the XR stream.

Within the context of a present disclosure, an “XR experience” isunderstood to be a presentation of XR media. For instance, an XRexperience could comprise a multi-player XR game, a virtual tour (e.g.,of a museum, historical site, real estate, or the like), a trainingsimulation (e.g., for an emergency responder, a vehicle operator, or thelike), a meeting (e.g., for professional or educational purposes), animmersive film presentation, or another type of experience. An “XRstream” refers to a persistent record or file containing the datacorresponding to the XR experience (e.g., real and virtual video, audio,and other sensory modality files that, when presented to a user, createthe XR experience). The XR stream may comprise a prerecorded stream or adynamic stream that is recorded in real time as the XR experienceprogresses.

To better understand the present disclosure, FIG. 1 illustrates anexample network 100, related to the present disclosure. As shown in FIG.1 , the network 100 connects mobile devices 157A, 157B, 167A and 167B,and home network devices such as home gateway 161, set-top boxes (STBs)162A, and 162B, television (TV) 163, home phone 164, router 165,personal computer (PC) 166, immersive display 168, and so forth, withone another and with various other devices via a core network 110, awireless access network 150 (e.g., a cellular network), an accessnetwork 120, other networks 140 and/or the Internet 145. In someexamples, not all of the mobile devices and home network devices will beutilized in presentation of an XR experience with time shiftingcapabilities. For instance, in some examples, presentation of XR mediamay make use of the home network devices (e.g., immersive display 168and/or STB/DVR 162A), and may potentially also make use of anyco-located mobile devices (e.g., mobile devices 167A and 167B), but maynot make use of any mobile devices that are not co-located with the homenetwork devices (e.g., mobile devices 157A and 157B).

In one example, wireless access network 150 comprises a radio accessnetwork implementing such technologies as: global system for mobilecommunication (GSM), e.g., a base station subsystem (BSS), or IS-95, auniversal mobile telecommunications system (UMTS) network employingwideband code division multiple access (WCDMA), or a CDMA3000 network,among others. In other words, wireless access network 150 may comprisean access network in accordance with any “second generation” (2G),“third generation” (3G), “fourth generation” (4G), Long Term Evolution(LTE) or any other yet to be developed future wireless/cellular networktechnology including “fifth generation” (5G) and further generations.While the present disclosure is not limited to any particular type ofwireless access network, in the illustrative example, wireless accessnetwork 150 is shown as a UMTS terrestrial radio access network (UTRAN)subsystem. Thus, elements 152 and 153 may each comprise a Node B orevolved Node B (eNodeB).

In one example, each of mobile devices 157A, 157B, 167A, and 167B maycomprise any subscriber/customer endpoint device configured for wirelesscommunication such as a laptop computer, a Wi-Fi device, a PersonalDigital Assistant (PDA), a mobile phone, a smartphone, an email device,a computing tablet, a messaging device, a wearable smart device (e.g., asmart watch or fitness tracker), a gaming console, and the like. In oneexample, any one or more of mobile devices 157A, 157B, 167A, and 167Bmay have both cellular and non-cellular access capabilities and mayfurther have wired communication and networking capabilities.

As illustrated in FIG. 1 , network 100 includes a core network 110. Inone example, core network 110 may combine core network components of acellular network with components of a triple play service network; wheretriple play services include telephone services, Internet services andtelevision services to subscribers. For example, core network 110 mayfunctionally comprise a fixed mobile convergence (FMC) network, e.g., anIP Multimedia Subsystem (IMS) network. In addition, core network 110 mayfunctionally comprise a telephony network, e.g., an InternetProtocol/Multi-Protocol Label Switching (IP/MPLS) backbone networkutilizing Session Initiation Protocol (SIP) for circuit-switched andVoice over Internet Protocol (VoIP) telephony services. Core network 110may also further comprise a broadcast television network, e.g., atraditional cable provider network or an Internet Protocol Television(IPTV) network, as well as an Internet Service Provider (ISP) network.The network elements 111A-111D may serve as gateway servers or edgerouters to interconnect the core network 110 with other networks 140,Internet 145, wireless access network 150, access network 120, and soforth. As shown in FIG. 1 , core network 110 may also include aplurality of television (TV) servers 112, a plurality of content servers113, a plurality of application servers 114, an advertising server (AS)117, and an extended reality (XR) server 115 (e.g., an applicationserver). For ease of illustration, various additional elements of corenetwork 110 are omitted from FIG. 1 .

In one example, XR server 115 may generate computer-generated content(e.g., digital overlays which may be combined with a live mediaincluding images of a “real world” environment, or entirely digitalenvironments) to produce an extended reality experience. For instance,where the computer-generated content comprises a digital overlay, thecomputer-generated content may include renderings of virtual objectsthat do not exist in the real world environment, such as graphics, text,audio clips, and the like. However, when the computer-generated contentis synchronized with the live footage of the “real world” environment onan immersive display (e.g., over a live video stream on a television oron a live view through a head mounted display), it may appear to a userthat the virtual objects are present in the “real world” environment.

Where the computer-generated content is an entirely digital environment,the entirely digital environment may appear to the user as a simulatedenvironment in which the user may interact with objects and/or otherusers. In one example, the extended reality experience for which thecomputer-generated content is rendered is a multi-user experience, suchas a multi-player or cooperative video game, an immersive filmpresentation, a training simulation, a virtual tour or meeting, and/orother types of experience. Collectively, the computer-generated contentand any real-world content with which the computer-generated content iscombined may comprise an XR stream, e.g., a persistent record or filecontaining the data corresponding to the XR experience.

In one example the XR server 115 may collect data provided by users ofthe network 100. The collected data may be provided directly to the XRserver 115 by the users, e.g., via the mobile devices 157A, 157B, 167A,and 167B, the PC 166, the home phone 164, the TV 163, and/or theimmersive display 168. Alternatively, the data may be retrieved fromnetwork storage, e.g., application servers 114, by the XR server 115.For instance the collected data may comprise user profiles maintained bya network service (e.g., an Internet service provider, a streaming mediaservice, a gaming subscription, etc.), portions of social media profilesmaintained by a social media web site (e.g., a social networking site, ablogging site, a photo-sharing site, etc.). The data may indicateinformation about the users, such as the users' virtual or realidentities, avatars for use in the XR experience, and the like. The XRserver 115 may also have access to third party data sources (e.g.,server 149 in other network 140), where the third party data sources maystore the user data.

In a further example, the XR server 115 may generate a temporary,personal XR environment for a user, where the personal XR environmentmay be used for one or more of a plurality of purposes withoutdisrupting the experiences of other users in a multi-user XR experience.For instance, in one example, the personal XR environment may be used toacclimate a new user who is joining an in-progress XR experience byproviding a space in which the new user might review a summary of any ofthe XR experience that the new user missed (e.g., notable events, otherusers who are currently present in the XR experience, etc.). In anotherexample, the personal XR environment could similarly be used to ease auser's exit from an XR reality experience (e.g., by providing a summaryof the events that the user participated in, information about the realworld environment into which the user is exiting, etc.). In yet anotherexample, the personal XR environment may be used to provide a user witha space in which the user may freely explore or shift through the XRstream (e.g., moving backward to a previous time in the XR stream,forward to a later time in the XR stream, etc.). In each case, thepersonal XR environment provides a space for a user to freely exploreand acclimate without affecting what other users see or experience.Thus, the personal XR environment may help to catch late joining usersup to speed and ease the late joining users' integration into the XRexperience.

The XR server 115 may interact with television servers 112, contentservers 113, and/or advertising server 117, to select which videoprograms, or other content and advertisements, if any, to include in anXR experience. For instance, the content servers 113 may store scheduledtelevision broadcast content for a number of television channels,video-on-demand programming, local programming content, gaming content,and so forth. Alternatively, or in addition, content providers maystream various contents to the core network for distribution to varioussubscribers, e.g., for live content, such as news programming, sportingevents, and the like. In one example, advertising server 117 stores anumber of advertisements that can be selected for presentation to users,e.g., in the home network 160 and at other downstream viewing locations.For example, advertisers may upload various advertising content to thecore network 110 to be distributed to various users. Any of the contentstored by the television servers 112, content servers 113, and/oradvertising server 117 may be used to generate computer-generatedcontent which, when presented alone or in combination with pre-recordedor real-world content or footage, produces an XR experience.

In one example, any or all of the television servers 112, contentservers 113, application servers 114, XR server 115, and advertisingserver 117 may comprise a computing system, such as computing system 300depicted in FIG. 3 .

In one example, the access network 120 may comprise a Digital SubscriberLine (DSL) network, a broadband cable access network, a Local AreaNetwork (LAN), a cellular or wireless access network, a 3^(rd) partynetwork, and the like. For example, the operator of core network 110 mayprovide a cable television service, an IPTV service, or any other typesof television service to subscribers via access network 120. In thisregard, access network 120 may include a node 122, e.g., a mini-fibernode (MFN), a video-ready access device (VRAD) or the like. However, inanother example node 122 may be omitted, e.g., for fiber-to-the-premises(FTTP) installations. Access network 120 may also transmit and receivecommunications between home network 160 and core network 110 relating tovoice telephone calls, communications with web servers via the Internet145 and/or other networks 140, and so forth.

Alternatively, or in addition, the network 100 may provide televisionservices to home network 160 via satellite broadcast. For instance,ground station 130 may receive television content from televisionservers 112 for uplink transmission to satellite 135. Accordingly,satellite 135 may receive television content from ground station 130 andmay broadcast the television content to satellite receiver 139, e.g., asatellite link terrestrial antenna (including satellite dishes andantennas for downlink communications, or for both downlink and uplinkcommunications), as well as to satellite receivers of other subscriberswithin a coverage area of satellite 135. In one example, satellite 135may be controlled and/or operated by a same network service provider asthe core network 110. In another example, satellite 135 may becontrolled and/or operated by a different entity and may carrytelevision broadcast signals on behalf of the core network 110.

In one example, home network 160 may include a home gateway 161, whichreceives data/communications associated with different types of media,e.g., television, phone, and Internet, and separates thesecommunications for the appropriate devices. The data/communications maybe received via access network 120 and/or via satellite receiver 139,for instance. In one example, television data is forwarded to set-topboxes (STBs)/digital video recorders (DVRs) 162A and 162B to be decoded,recorded, and/or forwarded to television (TV) 163 and/or immersivedisplay 168 for presentation. Similarly, telephone data is sent to andreceived from home phone 164; Internet communications are sent to andreceived from router 165, which may be capable of both wired and/orwireless communication. In turn, router 165 receives data from and sendsdata to the appropriate devices, e.g., personal computer (PC) 166,mobile devices 167A and 167B, and so forth. In one example, router 165may further communicate with TV (broadly a display) 163 and/or immersivedisplay 168, e.g., where one or both of the television and the immersivedisplay incorporate “smart” features. In one example, router 165 maycomprise a wired Ethernet router and/or an Institute for Electrical andElectronics Engineers (IEEE) 802.11 (Wi-Fi) router, and may communicatewith respective devices in home network 160 via wired and/or wirelessconnections.

It should be noted that as used herein, the terms “configure” and“reconfigure” may refer to programming or loading a computing devicewith computer-readable/computer-executable instructions, code, and/orprograms, e.g., in a memory, which when executed by a processor of thecomputing device, may cause the computing device to perform variousfunctions. Such terms may also encompass providing variables, datavalues, tables, objects, or other data structures or the like which maycause a computer device executing computer-readable instructions, code,and/or programs to function differently depending upon the values of thevariables or other data structures that are provided. For example, oneor both of the STB/DVR 162A and STB/DVR 162B may host an operatingsystem for presenting a user interface via TVs 163 and/or immersivedisplay 168, respectively. In one example, the user interface may becontrolled by a user via a remote control or other control devices whichare capable of providing input signals to a STB/DVR. For example, mobiledevice 167A and/or mobile device 167B may be equipped with anapplication to send control signals to STB/DVR 162A and/or STB/DVR 162Bvia an infrared transmitter or transceiver, a transceiver for IEEE802.11 based communications (e.g., “Wi-Fi”), IEEE 802.15 basedcommunications (e.g., “Bluetooth”, “ZigBee”, etc.), and so forth, whereSTB/DVR 162A and/or STB/DVR 162B are similarly equipped to receive sucha signal. Although STB/DVR 162A and STB/DVR 162B are illustrated anddescribed as integrated devices with both STB and DVR functions, inother, further, and different examples, STB/DVR 162A and/or STB/DVR 162Bmay comprise separate STB and DVR components.

Those skilled in the art will realize that the network 100 may beimplemented in a different form than that which is illustrated in FIG. 1, or may be expanded by including additional endpoint devices, accessnetworks, network elements, application servers, etc. without alteringthe scope of the present disclosure. For example, core network 110 isnot limited to an IMS network. Wireless access network 150 is notlimited to a UMTS/UTRAN configuration. Similarly, the present disclosureis not limited to an IP/MPLS network for VoIP telephony services, or anyparticular type of broadcast television network for providing televisionservices, and so forth.

To further aid in understanding the present disclosure, FIG. 2illustrates a flowchart of a method 200 for presenting an XR experiencewith time shifting capabilities, in accordance with the presentdisclosure. In one example, the method 200 may be performed by an XRserver that is configured to generate computer-generated content thatmay be blended or synchronized with a “real world” environment toproduce an extended reality experience, such as the XR server 115illustrated in FIG. 1 . However, in other examples, the method 200 maybe performed by another device, such as the processor 302 of the system300 illustrated in FIG. 3 .

The method 200 begins in step 202. In step 204, the processing systemmay present an extended reality (XR) experience to a plurality of userdevices, where the presenting includes presenting a time control alongwith an XR stream. In one example, the XR stream may comprise a virtualoverlay that each user device of the plurality of user devices rendersfor superimposition over a respective view of the real world. In anotherexample, the XR stream may comprise an entirely virtual environment thateach user device renders for presentation in place of a respective viewof the real world.

As discussed above, the XR experience may comprise a multi-player videogame, a virtual tour, a meeting, an immersive film, or another type ofXR experience. The XR stream, as discussed above, comprises a persistentrecord of the data corresponding to the XR experience. In one example,the processing system may begin recording and storing the XR stream assoon as the XR experience begins and may save the XR stream for at leastsome period of time after all of the users of the user devices havecompleted or exited the XR experience. In one example, the plurality ofuser devices may include one or more of: an immersive display, a mobilephone, a computing device, or any other device that is capable ofrendering an XR environment.

The time control comprises a control that allows a user device to movebackward and forward in the XR stream, on demand. For instance, if auser uses the user device to join the XR experience when the XRexperience is already in progress (e.g., join a game after the users ofthe rest of the user devices have been playing for several minutes orhours, join a virtual tour that is already in progress, etc.), the userdevice may be able to move backward in time (e.g., rewind) in the XRstream to see what happened earlier in the XR experience.

The user device may also move forward in time (e.g., fast forward) inthe XR stream. If the XR stream is a pre-recorded stream, then the userdevice may be able to move forward in time anywhere in the XR stream(e.g., may skip ahead directly to a desired portion of the XR stream).If the XR stream is not a pre-recorded XR stream or is a real-timestream, then the user device may be able to move forward in time up tothe current time (e.g., after moving backward to experience an earlierpart of the XR stream, the user device may move forward to catch back upto the current time). It should be noted that even users who have notjoined the XR experience late may still wish to move backward and/orforward in the XR stream. For instance, a user may need to refresh hismemory about something that happened or something that was said earlierin the XR experience, or may simply wish to revisit an earlier event inthe XR experience that was particularly enjoyable or informative (e.g.,reviewing an earlier disclosed clue or reviewing an earlier action takenby a character in the XR experience).

In one example, the copy of the XR stream may include markers toindicate where certain notable events occurred in the XR experience. Themarkers may serve as bookmarks that allow user devices to move directlyto specific points in time in the XR stream, rather than having tosearch through the entire XR stream to find the events. For instance, ifthe XR experience comprises a multi-player battle game, then a marker inthe corresponding XR stream might designate a specific battle that theusers participated in or specific points in the specific battle wherespecific objectives were achieved. If the XR experience is a virtualmuseum tour, a marker in the corresponding XR stream might designate aspecific exhibit in the museum. In some examples, a marker may designatea synchronization point for all user devices (e.g., all user devicesmust jump to the marker at a specific time) or a mandatory event thatall user devices must render (e.g., a game tutorial that users mustreview before playing an XR game).

In one example, the time control may be presented as a graphical elementthat can be manipulated by a user of a user device to control theprogress of the XR stream on the user device. For instance, in oneexample, the graphical element comprises a clock, where the user canadjust the hands on the clock or adjust the number displayed on theclock to indicate a specific time point in the XR stream. In anotherexample, the graphical element comprises one or more thumbnail imagesextracted from corresponding time points in the XR stream. In anotherexample, the graphical element comprises a slider that the user can movebackward or forward to indicate a magnitude and direction of movementthrough the XR stream (e.g., ten minutes backward, 2 minutes forward,etc.).

In further examples, however, the time control may comprise anon-graphical control. For instance, the time control could be a voiceoperable control that responds to verbal commands from a user of a userdevice (e.g., “go back five minutes,” “move forward at double speed,”etc.). In another example, the time control could monitor user gesturesfor gestures that are mapped to specific actions (e.g., moving theuser's hand to the left indicates moving backward, moving the user'shand to the right indicates moving forward, holding the user's hand palmforward indicates pausing, etc.).

In optional step 206 (illustrated in phantom), the processing system mayreceive, while the XR experience is in progress, a first signal from afirst user device of the plurality of user devices, where the firstsignal indicates that a user of the first user device wishes to join theXR experience. For instance, although the XR experience is already inprogress, the user of the first user device may still wish to join late.

In optional step 208 (illustrated in phantom), the processing system maypresent a personal XR environment to the first user device, in responseto the first signal, where the personal XR environment facilitates atransition of the user of the first user device into the XR experiencewhile the XR experience continues to be presented to the other userdevices of the plurality of devices. In other words, the personal XRenvironment is presented to the first user device concurrently with theXR experience being presented to the other user devices.

As discussed above, the personal XR environment may facilitate atransition of the user of the first user device into the XR experience.For instance, as discussed above, the user may be seeking to join an XRexperience that is already in progress; as such, the user would havemissed out on any portion of the XR experience that occurred prior tothe user joining. Thus, in one example, the personal XR environment mayprovide information about the portion of the XR experience that the userof the first user device missed (e.g., a recap or summary of events thathave occurred, an indication of how long the XR experience has beenpresented, the real or virtual identities of users who are currentlyparticipating in the XR experience, and/or other information). Thus, thepersonal XR environment may be considered a virtual “green room,” wherethe user of the first user device can acclimate to the XR experience andbe “caught up to speed” without disrupting the experiences of otherusers.

In optional step 210 (illustrated in phantom), the processing system maypresent a notification to the other devices of the plurality of deviceswhile presenting the XR experience (e.g., at the time point in the XRexperience that is currently being rendered) to the first user device,where the notification indicates that the user of the first device hasjoined the XR experience. In one example, the notification may comprisean instruction that causes a user device to present at least one of: avisual notification, an audible notification, and a tactilenotification. For instance, a visual notification might comprise aticker, an icon, or the like that may be presented on a display of auser device that indicates that the user of the first user device hasjust joined. An audible notification might comprise a beep, anannouncement (e.g., synthesized speech), or the like that may bepresented via a speaker of a user device that indicates that the user ofthe first user device has just joined. A tactile notification mightcomprise a rumble, a shake, or the like that may be presented via ahaptic feedback mechanism of a user device that indicates that the userof the first user device has just joined.

In step 212, the processing system may receive a second signal from thefirst user device of the plurality of user devices via the time control,where the second signal indicates that the user of the first user devicewishes to shift the XR stream to a first point in time that is differentfrom a time that is currently being rendered in the XR stream (e.g.,such that the first user device presents the first point in time, ratherthan the current time). For instance, the user of the first user devicemay wish to shift backward to an earlier time point in the XR stream andmanipulate the time control, described above, to indicate the desire toshift backward and/or by how much the user wishes to shift backward(e.g., “go back ten minutes” or “go back to Event X”). In anotherexample, specific markers may be automatically created by the processingsystem for significant events within the XR stream. In one example of animmersive film presentation, the processing system may create markers atspecific scenes, specific actor appearances, or specific songinclusions, all detected by an automated recognition system, that arepresented to the first user for jumping to these instances in the XRstream.

In step 214, the processing system may present the personal XRenvironment to the first user device, in response to the second signal,where the personal XR environment presents the first point in time inthe XR stream to the first user device without changing the time pointof the XR stream that is currently being presented to the other userdevices of the plurality of user devices. In other words, the personalXR environment is only presented to the first user device concurrentlywith the current XR stream being presented to the other user devices(e.g., as in step 208). In another example, a user of a second userdevice may request to shift the XR stream ahead to a later point intime. For instance, the user of the second user device may want to skipover a certain part of the XR stream, or skip ahead to a well-knownportion of the XR stream (e.g., the performance of a popular song).Notification of the shift for the second user device could be providedin real time to the first user device.

In optional step 216 (illustrated in phantom), the processing system maymodify the XR stream to indicate that the first user device is notpresenting the time that is currently being rendered in the XR stream(or, in other words, to indicate that the user of the first user deviceis not “present” in the time that is currently being rendered in the XRstream). For instance, a visual modification to the XR stream mightcomprise blurring, fading, or graying out an avatar or otherrepresentations of the user of the first user device in the XR stream toindicate that the user is not “present” in the time that is currentlybeing rendered. In another example, the avatar or other representationsmay be replaced with a placeholder or icon that specifically indicatesthat the user is not present in the time that is currently beingrendered. In one example, the processing system “moves” the first userdevice to a personal XR environment at a different time instance in theXR stream, but a visual rendering of the first user device'slast-attended instance in time in the XR stream (e.g., “the present” or“real time”) may be visualized (e.g., as a picture in a picture) for theuser of the first user device. In one example, the visualization assumesa scene-level (e.g., zoomed out) third person perspective rendering ofthe last-attended instance in time. In another example, activation ofthe user of the first user's device avatar, the visualization of thelast-attended instance in time, or other visual modifications may betemporary and triggered by a specific event in either time instance(last-attended or current) within the XR stream. For example, no avataror picture in picture display may be presented until a second user is“attacked” by a virtual monster in an XR game.

In another example, an audio modification to the XR stream mightcomprise presenting the voice of the user of the first user device as avoiceover, rather than coming from the user's avatar. Alternatively, anaudio modification might include presenting the voice of the user of thefirst user device at a lower volume or with some added audio effect(e.g., an echo, reverb, or other effects). In another example,interactive conversations that are happening in real time between afirst user and a second user may be created and facilitated by theprocessing system. In one example, in response to the first usershifting to a personal XR environment, a second user may speak, gesture,or type, “When are we going to dinner?” Although the first user and thesecond user are in different personal experiences within the XR stream,(the time order may not be relevant), the first user may be presentedwith an audio message, a visible avatar speaking, a text message, orsome other indications that the conversation is live and not part of theshifted XR stream. In one example, this conversation between the firstuser and the second user may also become a specific marker in the XRstream at the respective time instances of the first user and the seconduser. In another example, the conversation may be marked as ephemeraland/or confidential, and the addition of the conversation to the XRstream may be suppressed.

In optional step 218 (illustrated in phantom), the processing system mayreceive a third signal from the first user device indicating that thefirst user device wishes to exit the XR experience. In one example, theXR experience may still be in progress (i.e., the first user device isrequesting to exit before the XR experience has been completed). Inanother example, however, the XR experience may be coming to a plannedend when the third signal is received (e.g., the XR experience may havebeen configured to last for a predefined period of time, until the userscomplete a predefined set of tasks, or the like).

In optional step 220 (illustrated in phantom), the processing system maypresent the personal XR environment to the first user device, inresponse to the third signal, where the personal XR environmentfacilitates a transition of the user of the first user device out of theXR experience. For instance, the personal XR environment may present theuser with a summary of the user's activities in the XR experience, givethe user an opportunity to remotely save a portion of the XR experienceor receive further information or communications relating to the XRexperience, give the user an opportunity to “bookmark” their place inthe XR experience to return later, or the like.

Where the third signal is received while the XR experience is still inprogress, the processing system may continue to present the XRexperience to the other user devices of the plurality of user deviceswhile presenting the personal XR environment to the first user device.In other words, the personal XR environment is presented to the firstuser device concurrently with the XR experience being presented to theother user devices.

However, where the third signal is received along with signals from theother user devices (e.g., at a time where the XR experience is plannedto end or upon users of the plurality of devices agreeing to end the XRexperience), the processing system may present the personal XRenvironment to all of the user devices, in order to synchronize theplurality of user devices prior to ending the XR experience. This mayallow all of the users of the plurality of user devices to share acommon end to the XR experience.

The method 200 may end in step 222.

Thus, the method 200 may allow users who join XR experiences “late” toexperience portions of the XR experience that the users may have missed(e.g., by moving backward in the XR stream), without disrupting theexperiences of other users who may not have missed anything. The method200 may also allow users, where available, to skip portions of an XRexperience (e.g., to move forward in the XR stream), also withoutdisrupting the experiences of other users.

In some examples, the XR experience may be configured, prior topresentation (and in response to user-selected options), to enable ordisable the features described above. For instance, users of the XRexperience may agree, before beginning the XR experience, whether or notto allow other users to join after the XR experience begins, leavebefore the XR experience ends, or shift in time through the XR streamwhile the XR experience is in progress. In further examples, the usersmay agree on specific events within the XR stream that may be bookmarkedto allow users to revisit (i.e., where users may not be able to shift intime through the XR stream to events or times that are not bookmarked).

In a further example still, the processing system may automaticallydetermine whether to enable or disable the features described above. Forinstance, the processing system may monitor the XR experience and maydetermine whether certain features would enhance or detract from the XRexperience based on changes in the XR environment. As an example, duringa virtual conference or meeting, the processing may determine that usersshould be allowed to join the XR experience late, but should not be ableto move backward or forward through the XR stream. In further examples,the processing system may adapt the manner in which objects, avatars, orcameras move in the XR experience in response to a user's movement.

Examples of the present disclosure may therefore prove beneficial invarious fields of use. In a simple example, a user could be walkingaround with a camera (e.g., filming a popular tourist destination) andstreaming to the user's friends or family. Examples of the presentdisclosure would allow those friends and family to see any portion ofthe stream that they may have missed.

In another example, a tour (e.g., of a museum, a historical site, or thelike) could be conducted at least partially virtually. For instance, thetour could be conducted exclusively for the benefit of remoteparticipants or for a combination of in-person participants and remoteparticipants. The tour leader may walk around with a camera and providecommentary. Remote participants may be able to join the tour after thetour has begun, and still skip back to see anything that may have beenmissed. The tour leader (and potentially other participants) may benotified when remote participants join or leave the tour.

In another example, a trial lawyer may walk a jury through a virtualrecreation of a crime scene while providing commentary. The trial lawyermay be able to jump the presentation of the virtual recreation backwardor forward to specific events that the jury needs to see.

In another example, the present disclosure may allow differentviewpoints of the same event (e.g., a live event or a virtual event) tobe merged to create a single common narrative that pulls from thedifferent viewpoints and/or synchronizes the experiences of the usersassociated with the multiple viewpoints. For instance, the event may bea concert. One “attendee” may be able to go back in the XR stream of theconcert to watch the band perform his favorite song again. However, theattendee may still be able to talk, in real time, with his friend who iswatching the song the band is currently playing or who has time shiftedto watch a different song.

Moreover, although the personal XR environment described above isdescribed as being used to ease transitions into and out of the XRexperience, the personal XR environment could also be used to easetransitions related to time shifting. For instance, if a user hasshifted backward in time for a significant period (e.g., more than xminutes), then attempts to jump ahead to the current time point, thepersonal XR environment could serve to ease the user's transition backto the current time point. In one example, easing the transition maycomprise adjusting the speed with which the user moves ahead in the XRstream. For instance, the speed may gradually slow as the user'slocation in the XR stream approaches the current time point, so that themovement through the XR stream does not abruptly stop when the currenttime point is reached.

Although not expressly specified above, one or more steps of the method200 may include a storing, displaying and/or outputting step as requiredfor a particular application. In other words, any data, records, fields,and/or intermediate results discussed in the method can be stored,displayed and/or outputted to another device as required for aparticular application. Furthermore, operations, steps, or blocks inFIG. 2 that recite a determining operation or involve a decision do notnecessarily require that both branches of the determining operation bepracticed. In other words, one of the branches of the determiningoperation can be deemed as an optional step. However, the use of theterm “optional step” is intended to only reflect different variations ofa particular illustrative embodiment and is not intended to indicatethat steps not labelled as optional steps to be deemed to be essentialsteps. Furthermore, operations, steps or blocks of the above describedmethod(s) can be combined, separated, and/or performed in a differentorder from that described above, without departing from the examples ofthe present disclosure.

FIG. 3 depicts a high-level block diagram of a computing devicespecifically programmed to perform the functions described herein. Forexample, any one or more components or devices illustrated in FIG. 1 ordescribed in connection with the method 200 may be implemented as thesystem 300. For instance, a server (such as might be used to perform themethod 200) could be implemented as illustrated in FIG. 3 .

As depicted in FIG. 3 , the system 300 comprises a hardware processorelement 302, a memory 304, a module 305 for presenting an XR experiencewith time shifting capabilities, and various input/output (I/O) devices306.

The hardware processor 302 may comprise, for example, a microprocessor,a central processing unit (CPU), or the like. The memory 304 maycomprise, for example, random access memory (RAM), read only memory(ROM), a disk drive, an optical drive, a magnetic drive, and/or aUniversal Serial Bus (USB) drive. The module 305 for presenting an XRexperience with time shifting capabilities may include circuitry and/orlogic for performing special purpose functions relating to the operationof a home gateway or AR server. The input/output devices 306 mayinclude, for example, a camera, a video camera, storage devices(including but not limited to, a tape drive, a floppy drive, a hard diskdrive or a compact disk drive), a receiver, a transmitter, a speaker, adisplay, a speech synthesizer, an output port, and a user input device(such as a keyboard, a keypad, a mouse, and the like), or a sensor.

Although only one processor element is shown, it should be noted thatthe computer may employ a plurality of processor elements. Furthermore,although only one computer is shown in the Figure, if the method(s) asdiscussed above is implemented in a distributed or parallel manner for aparticular illustrative example, i.e., the steps of the above method(s)or the entire method(s) are implemented across multiple or parallelcomputers, then the computer of this Figure is intended to representeach of those multiple computers. Furthermore, one or more hardwareprocessors can be utilized in supporting a virtualized or sharedcomputing environment. The virtualized computing environment may supportone or more virtual machines representing computers, servers, or othercomputing devices. In such virtualized virtual machines, hardwarecomponents such as hardware processors and computer-readable storagedevices may be virtualized or logically represented.

It should be noted that the present disclosure can be implemented insoftware and/or in a combination of software and hardware, e.g., usingapplication specific integrated circuits (ASIC), a programmable logicarray (PLA), including a field-programmable gate array (FPGA), or astate machine deployed on a hardware device, a computer or any otherhardware equivalents, e.g., computer readable instructions pertaining tothe method(s) discussed above can be used to configure a hardwareprocessor to perform the steps, functions and/or operations of the abovedisclosed method(s). In one example, instructions and data for thepresent module or process 305 for presenting an XR experience with timeshifting capabilities (e.g., a software program comprisingcomputer-executable instructions) can be loaded into memory 304 andexecuted by hardware processor element 302 to implement the steps,functions or operations as discussed above in connection with theexample method 200. Furthermore, when a hardware processor executesinstructions to perform “operations,” this could include the hardwareprocessor performing the operations directly and/or facilitating,directing, or cooperating with another hardware device or component(e.g., a co-processor and the like) to perform the operations.

The processor executing the computer readable or software instructionsrelating to the above described method(s) can be perceived as aprogrammed processor or a specialized processor. As such, the presentmodule 305 for presenting an XR experience with time shiftingcapabilities (including associated data structures) of the presentdisclosure can be stored on a tangible or physical (broadlynon-transitory) computer-readable storage device or medium, e.g.,volatile memory, non-volatile memory, ROM memory, RAM memory, magneticor optical drive, device or diskette and the like. More specifically,the computer-readable storage device may comprise any physical devicesthat provide the ability to store information such as data and/orinstructions to be accessed by a processor or a computing device such asa computer or an application server.

While various examples have been described above, it should beunderstood that they have been presented by way of example only, and notlimitation. Thus, the breadth and scope of a preferred example shouldnot be limited by any of the above-described example examples, butshould be defined only in accordance with the following claims and theirequivalents.

1.-20. (canceled)
 21. A method comprising: presenting, by a processingsystem including at least one processor, an extended reality experienceto a plurality of user devices, wherein the presenting includespresenting a time control along with an extended reality stream, whereinthe extended reality stream comprises a dynamic stream that is recordedin real time as the extended reality experience progresses; receiving,by the processing system, a first signal from a first user device of theplurality of user devices via the time control, wherein the first signalindicates that a user of the first user device wishes to shift theextended reality stream to a first point in time that is different froma time that is currently being rendered in the extended reality stream;presenting, by the processing system, a personal extended realityenvironment to the first user device in response to the first signal,wherein the personal extended reality environment presents the firstpoint in time in the extended reality stream to the first user devicewithout changing a time point of the extended reality stream that iscurrently being presented to other user devices of the plurality of userdevices; and modifying, by the processing system, a representation ofthe first user in the extended reality stream as presented to the otheruser devices of the plurality of user devices to indicate that the firstuser device is not presenting the time that is currently being renderedin the extended reality stream.
 22. The method of claim 21, wherein thetime control comprises a control that, when manipulated by a user of anyuser device of the plurality of user devices, allows the any user deviceto move backward and forward in time through the extended reality streamon demand.
 23. The method of claim 22, wherein the time control ispresented as at least one of: a clock, a series of thumbnail imagesextracted from a series of corresponding time points in the extendedreality stream, or a slider.
 24. The method of claim 22, wherein thetime control is responsive to a user gesture.
 25. The method of claim22, wherein the time control is responsive to a typed input.
 26. Themethod of claim 21, wherein the extended reality stream is annotatedwith markers that indicate where notable events occurred in the extendedreality stream.
 27. The method of claim 26, wherein the first signalindicates a selection by the first user device of one of the markers.28. The method of claim 26, wherein at least one marker of the markersdesignates a synchronization point for all user devices of the pluralityof user devices.
 29. The method of claim 21, wherein the modifyingcomprises visually altering an avatar of the representation of the userin the extended reality stream that represents the user of the firstuser device.
 30. The method of claim 29, wherein the visually alteringcomprises at least one of: blurring the avatar, fading the avatar,graying out the avatar, or replacing the avatar with a placeholder. 31.The method of claim 21, wherein the modifying comprises presenting avoice of the representation of the user of the first user device as avoiceover in the extended reality stream.
 32. The method of claim 21,wherein the modifying comprises adding an audio effect to a voice of therepresentation of the user of the first user device.
 33. The method ofclaim 21, further comprising: receiving, by the processing system afterpresenting the extended reality experience but prior to receiving thefirst signal, a second signal from the first user device, wherein thesecond signal indicates that the user of first user device wishes tojoin the extended reality experience that is in progress; andpresenting, by the processing system, the personal extended realityenvironment to the first user device in response to the second signal,wherein the personal extended reality environment facilitates atransition of the user of the first user device into the extendedreality experience while the extended reality experience continues to bepresented to the other user devices of the plurality of user devices.34. The method of claim 33, wherein when the personal extended realityenvironment is presented in response to the second signal, the personalextended reality environment provides information about a portion of theextended reality experience that the user of the first user devicemissed.
 35. The method of claim 21, further comprising: receiving, bythe processing system after the first user device has joined theextended reality experience, a second signal from the first user device,wherein the second signal indicates that the user of first user devicewishes to exit the extended reality experience; and presenting, by theprocessing system, the personal extended reality environment to thefirst user device in response to the second signal, wherein the personalextended reality environment facilitates a transition of the user of thefirst user device out of the extended reality experience.
 36. The methodof claim 35, wherein when the second signal is received while theextended reality experience is still in progress, the processing systemcontinues to present the extended reality experience to the other userdevices of the plurality of user devices, concurrently with presentingthe personal extended reality environment to the first user device. 37.The method of claim 21, wherein the first point in time is one of aplurality of points in time in the extended reality stream that arepredefined with bookmarks in order to allow users of the plurality ofuser devices to revisit the plurality of points in time.
 38. The methodof claim 21, wherein the extended reality experience is at least oneexperience of: a multi-player video game, an immersive filmpresentation, a virtual tour, a virtual meeting, a training simulation,or a virtual recreation of a real world environment.
 39. Anon-transitory computer-readable medium storing instructions which, whenexecuted by a processing system, cause the processing system to performoperations, the operations comprising: presenting an extended realityexperience to a plurality of user devices, wherein the presentingincludes presenting a time control along with an extended realitystream, wherein the extended reality stream comprises a dynamic streamthat is recorded in real time as the extended reality experienceprogresses; receiving a first signal from a first user device of theplurality of user devices via the time control, wherein the first signalindicates that a user of the first user device wishes to shift theextended reality stream to a first point in time that is different froma time that is currently being rendered in the extended reality stream;presenting a personal extended reality environment to the first userdevice in response to the first signal, wherein the personal extendedreality environment presents the first point in time in the extendedreality stream to the first user device without changing a time point ofthe extended reality stream that is currently being presented to otheruser devices of the plurality of user devices; and modifying arepresentation of the first user in the extended reality stream aspresented to the other user devices of the plurality of user devices toindicate that the first user device is not presenting the time that iscurrently being rendered in the extended reality stream.
 40. A devicecomprising: a processor; and a computer-readable medium storinginstructions which, when executed by the processor, cause the processorto perform operations, the operations comprising: presenting an extendedreality experience to a plurality of user devices, wherein thepresenting includes presenting a time control along with an extendedreality stream, wherein the extended reality stream comprises a dynamicstream that is recorded in real time as the extended reality experienceprogresses; receiving a first signal from a first user device of theplurality of user devices via the time control, wherein the first signalindicates that a user of the first user device wishes to shift theextended reality stream to a first point in time that is different froma time that is currently being rendered in the extended reality stream;presenting a personal extended reality environment to the first userdevice in response to the first signal, wherein the personal extendedreality environment presents the first point in time in the extendedreality stream to the first user device without changing a time point ofthe extended reality stream that is currently being presented to otheruser devices of the plurality of user devices; and modifying arepresentation of the first user in the extended reality stream aspresented to the other user devices of the plurality of user devices toindicate that the first user device is not presenting the time that iscurrently being rendered in the extended reality stream.